Testing software systems of organizations is a process that needs to be frequently performed in order to validate the systems' performance. This is especially true when new software modules are installed and/or when software modules undergo changes such as when they are updated or customized. Such software systems (e.g., SAP ERP, or Oracle EBS) are often large and require running of many tests for their validation. Consequently, testing often ends up accounting for a large portion of the IT budget of the organizations.
Often, a software system belonging to an organization may be quite complex, containing a large number of software modules, and customized specifically for the organization (e.g., due to a selection of a specific set of software module and/or organization-specific customizations to software modules). However, software systems belonging to different organizations often utilize the same, or similar, software modules and may involve similar customizations. For example, software systems belonging to different organizations in the same field of operations may involve similar customizations that are related to the field of operations. Additionally, software systems of different organizations may involve similar updates, such as when a new build of a generic software module is released. Thus, despite organization-specific differences in software systems of different organizations, it is often the case that testers belonging to the different organizations end up running the same, or quite similar, tests on their respective systems.
Despite the fact that different organizations often end up utilizing similar tests, it is often the case that each organization develops its own testing suite. Thus, knowledge such as best practices and known system vulnerabilities basically need to be rediscovered by each organization. Thus, the cumulative testing experience that may be aggregated from different organizations, the (testing) crowd wisdom, so to say, is typically not shared. Though it would be beneficial for organizations to share information regarding testing, there are several obstacles that hinder this process. First, software systems belonging to different organizations are often customized for their respective organizations. Thus, it is not likely that a test devised for a first organization will run without modification on a system belonging to a second organization. Additionally, tests used to test a system belonging to an organization often contain proprietary data related to the organization. However, organizations are not likely to share their testing data if it means that in the process, their proprietary data is at risk of being leaked to an unauthorized party.